Image pickup apparatus for acquiring a refocus image, method of controlling image pickup apparatus, and non-transitory computer-readable storage medium

ABSTRACT

An image pickup apparatus includes an imaging optical system, an image pickup element including a plurality of pixels, a lens array configured such that rays from the same position on an object plane are incident on pixels of the image pickup element different from each other depending on a pupil region of the imaging optical system through which the ray passes, an image processing unit configured to perform image processing for the input image acquired by the image pickup element to generate the output images, and a control unit configured to drive the imaging optical system to perform focus control, the image processing unit is configured to acquire information on a refocus control range, and the control unit is configured to perform the focus control based on the information on the refocus control range.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image pickup apparatus capable ofacquiring a refocus image.

2. Description of the Related Art

In recent years, image pickup apparatuses have been proposed whichperform a calculation for data acquired by an image pickup element anddigital image processing according to the calculation to output avariety of images. The literatures Ren Ng, et al., “Light FieldPhotography with a Hand-held Plenoptic Camera”, 2005 Computer ScienceTechnical Report CTSR, Todor Georgiev, et al., “Superresolution withPlenoptic 2.0 Camera”, 2009 Optical Society of America, and AaronIsaksen, et al., “Dynamically Reparameterized Light Fields”, ACMSIGGRAPH, pp. 297-306 (2000) disclose image pickup apparatuses thatsimultaneously acquire a two-dimensional intensity distribution of lightand angle information of a ray in an object space. The two-dimensionalintensity distribution of light and the angle information of the ray arehereinafter collectively referred to as a “light field”, and acquiringthe light field enables the acquisition of three-dimensional informationon an object space. The image pickup apparatuses described above canchange a focus position of an image, which is referred to as“refocusing”, change a shooting viewpoint, adjust a depth of field, andperform like operations, by acquiring the light field and performing theimage processing after shooting the image.

Japanese Patent Laid-Open No. (“JP”) 2011-109310 discloses aconfiguration which utilizes a refocus function as an auxiliary functionfor autofocusing. JP S59-146029 discloses a configuration which shootsan image with a focus position being displaced depending on a depth offield determined by an aperture stop such that a plurality of objectsare included within the depth of field.

While it is possible to change a focus position by the refocus functionafter a shooting, the focus position cannot be necessarily set to anarbitrary position because a range within which the focus position canbe changed is limited. This may prevent a user, after the shooting, fromchanging a focus position to another point intended by the user. Thismeans that a plurality of arbitrary objects cannot be included togetherin a refocus range with the configuration disclosed in JP 2011-109310.Similarly, it is impossible to generate a refocus image with theconfiguration disclosed in JP S59-146029.

SUMMARY OF THE INVENTION

The present invention provides an image pickup apparatus capable of,after a shooting, acquiring a refocus image with a focus positionintended by a user, a method of controlling the image pickup apparatus,and a non-transitory computer-readable storage medium.

An image pickup apparatus as one aspect of the present invention is animage pickup apparatus capable of reconstructing an input image togenerate a plurality of output images with focus positions differentfrom each other and includes an imaging optical system, an image pickupelement including a plurality of pixels, a lens array configured suchthat rays from the same position on an object plane are incident onpixels of the image pickup element different from each other dependingon a pupil region of the imaging optical system through which the raypasses, an image processing unit configured to perform image processingfor the input image acquired by the image pickup element to generate theoutput images, and a control unit configured to drive the imagingoptical system to perform focus control. The image processing unit isconfigured to acquire information on a refocus control range. Thecontrol unit is configured to perform the focus control based on theinformation on the refocus control range.

An image pickup apparatus as another aspect of the present invention isan image pickup apparatus capable of reconstructing an input image togenerate a plurality of output images with focus positions differentfrom each other and includes an imaging optical system, at least oneimage pickup element including a plurality of pixels, an imageprocessing unit configured to generate the output images from the inputimage acquired by the image pickup element, and a control unitconfigured to drive the imaging optical system to perform focus control.When a pupil of the imaging optical system is a pupil formed bycombining pupils of the plurality of optical systems, the plurality ofoptical systems are arranged such that rays from the same position on anobject plane are incident on pixels of the image pickup elementdifferent from each other depending on a pupil region of the imagingoptical system through which the ray passes. The image processing unitis configured to acquire information on a refocus control range. Thecontrol unit is configured to perform the focus control based on theinformation on the refocus control range.

A method of controlling an image pickup apparatus as another aspect ofthe present invention is a method of controlling an image pickupapparatus capable of reconstructing an input image to generate aplurality of output images with focus positions different from eachother and includes the steps of acquiring the input image that is animage formed by acquiring information on an object space from aplurality of viewpoints, by using an image pickup apparatus whichincludes an imaging optical system and an image pickup element includinga plurality of pixels, acquiring information on a refocus control range,and performing focus control based on the information on the refocuscontrol range.

A non-transitory computer-readable storage medium as another aspect ofthe present invention is a storage medium storing a program configuredto cause a computer to execute a method of controlling an image pickupapparatus capable of reconstructing an input image to generate aplurality of output images with focus positions different from eachother, and the method includes the steps of acquiring the input imagethat is an image formed by acquiring information on an object space froma plurality of viewpoints, by using an image pickup apparatus whichincludes an imaging optical system and an image pickup element includinga plurality of pixels, acquiring information on a refocus control range,and performing focus control based on the information on the refocuscontrol range.

Further features and aspects of the present invention will becomeapparent from the following description of exemplary embodiments withreference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an image pickup apparatus in first, second,and third embodiments.

FIG. 2 is a schematic configuration diagram of an image pickup unit inthe first embodiment.

FIG. 3 is a schematic configuration diagram of the image pickup unit inthe second embodiment.

FIG. 4 is a schematic configuration diagram of the image pickup unit inthe second embodiment.

FIG. 5 is a schematic configuration diagram of the image pickup unit inthe third embodiment.

FIG. 6 is a sectional view of the image pickup unit in the firstembodiment.

FIG. 7 is a sectional view of the image pickup unit in the secondembodiment.

FIG. 8 is a sectional view of an imaging optical system in the thirdembodiment.

FIG. 9 is a sectional view of an imaging optical system in a fourthembodiment.

FIGS. 10A and 10B are explanatory diagrams of generation of a refocusimage in the second embodiment.

FIG. 11 is an explanatory diagram of generation of a refocus image inthe first embodiment.

FIG. 12 is an explanatory diagram of a refocus control range in thefirst embodiment.

FIG. 13 is an explanatory diagram of a refocus control range in thesecond embodiment.

FIG. 14 is an explanatory diagram of a refocus control range in thethird embodiment.

FIG. 15 is a schematic configuration diagram of the image pickup unit inthe third embodiment.

FIG. 16 is a flowchart illustrating shooting processing in the first,second, third, and fourth embodiments.

FIG. 17 is a flowchart illustrating shooting processing in the first,second, third, and fourth embodiments.

FIG. 18 is a flowchart illustrating shooting processing in the first,second, third, and fourth embodiments.

FIGS. 19A and 19B are diagrams illustrating an example of a shootingscene in the first, second, third, and fourth embodiments.

FIGS. 20A and 20B are diagrams illustrating an example of a shootingscene in the first, second, third, and fourth embodiments.

FIGS. 21A and 21B are diagrams illustrating an example of a shootingscene in the first, second, third, and fourth embodiments.

FIGS. 22A and 22B are explanatory diagrams of the refocus control rangein the first embodiment.

FIG. 23 is a flowchart illustrating the shooting processing in thefirst, second, third, and fourth embodiments.

FIG. 24 is a flowchart illustrating the shooting processing in thefirst, second, third, and fourth embodiments.

FIG. 25 is a flowchart illustrating the shooting processing in thefirst, second, third, and fourth embodiments.

FIGS. 26A and 26B are diagrams illustrating an example of the shootingscene in the first, second, third, and fourth embodiments.

FIG. 27 is an optical arrangement diagram of the image pickup unit inthe first embodiment.

FIG. 28 is an explanatory diagram of the refocus control range in thefirst embodiment.

FIG. 29 is a flowchart of the shooting processing in the first, second,third, and fourth embodiments.

FIG. 30 is a flowchart of the shooting processing in the first, second,third, and fourth embodiments.

FIG. 31 is a block diagram of an image processing system in the fourthembodiment.

FIG. 32 is a schematic configuration diagram of the image processingsystem in the fourth embodiment.

FIG. 33 is a block diagram of a single viewpoint image acquiring unit inthe fourth embodiment.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described belowwith reference to the accompanied drawings. In each of the drawings, thesame elements will be denoted by the same reference numerals and theduplicate description thereof will be omitted.

An image pickup unit of this embodiment acquires a plurality of parallaximages, which are images of an object space shot from a plurality ofviewpoints, namely, a light field. The “object space” as used hereinmeans a real space in an angle of view that can be acquired by the imagepickup apparatus of this embodiment. Examples of the configuration ofthe image pickup unit include configurations illustrated in FIGS. 2 to 4in which a lens array is arranged at an image side of an imaging opticalsystem and a configuration illustrated in FIG. 5 in which a plurality ofimaging optical systems are arrayed. In addition, as an example of amethod of acquiring the light field, a method can be adopted whichperforms a shooting multiple times using an image pickup apparatus thatincludes the imaging optical system and an image pickup element whilechanging a position of the image pickup apparatus. In this method, theimage pickup apparatus captures parallax images in an object space attime points different from each other. For this reason, it is impossibleto acquire correct parallax information when a moving object is presentin the object space. It is thus desirable that the image pickup unit hasa configuration capable of simultaneously acquiring a plurality ofparallax images as illustrated in FIGS. 2 to 5.

Performing various processing such as extraction, change of order, andsynthesis of pixels, for the parallax images acquired with theconfigurations illustrated in FIGS. 2 to 5 enables refocusing, controlof a depth of field, change of a viewpoint, and the like. In thisembodiment, the processing is referred to as “reconstruction” and animage generated by the reconstruction as a “reconstructed image”. Inparticular, an image which has been subjected to refocus processing isreferred to as a “refocus image”. The refocus image may be an imagewhich has been subjected to processing such as noise reduction andreconstruction processing such as control of a depth of field. Inaddition, a range in an object space within which the refocusing can beperformed is referred to as a “refocus control range”.

A person or an object is not necessarily required to be present on anobject plane 201 illustrated in FIGS. 2 to 5. This is because a focusposition of the person or the object that is present at a far side or anear side of the object plane 201 can be controlled after a shooting bythe refocus processing. While each of the following embodiments may bedescribed using a one-dimensional system for ease of reference, asimilar description holds when a two-dimensional system is used.

First Embodiment

First of all, referring to FIGS. 1 and 2, a basic configuration of theimage pickup apparatus in the first embodiment of the present inventionwill be described. FIG. 1 is a block diagram of an image pickupapparatus 10 in this embodiment. FIG. 2 is a schematic configurationdiagram of an image pickup unit 100 in this embodiment.

The image pickup apparatus 10 of this embodiment is capable ofgenerating a plurality of output images with focus positions differentfrom each other by reconstructing an input image. As illustrated in FIG.2, the image pickup unit 100 includes an imaging optical system 101, alens array 102, and an image pickup element 103 which are arranged inthis order from the object plane 201. The details of the image pickupunit 100 will be described later.

The image pickup element 103 is a two-dimensional image pickup elementsuch as a CCD (Charge Coupled Device) or a CMOS (ComplementaryMetal-Oxide Semiconductor) sensor. The image pickup element 103 includesa plurality of pixels and photoelectrically converts an object image (anoptical image). In other words, energy of rays that have passed throughthe imaging optical system 101 and the lens array 102 and are thenincident on the image pickup element 103 is converted into an electricsignal (an analog signal). An A/D convertor 104 converts the analogsignal sent from the image pickup unit 100 (the image pickup element103) into a digital signal. An image processing unit 105 performspredetermined processing for the digital signal to generate a displayimage. In addition, the image processing unit 105 performs imagegeneration processing such as refocus processing, which will bedescribed later, based on image pickup data acquired by the image pickupunit 100 or on that acquired by a storage unit 109. As described above,the image processing unit 105 performs image processing for the imagepickup data, such as an input image, acquired by the image pickupelement 103 to generate the output images. The details of the imageprocessing unit 105 will be described later.

The display image generated by the image processing unit 105 is outputand displayed on a display unit 106 such as a liquid crystal display. Auser can perform the shooting while viewing the image being displayed onthe display unit 106 to check the image to be shot. In addition, thedisplay unit 106 may have a touch screen function. In this case, it isalso possible to treat a user's instruction given by using a touchscreen as an input. The display image is generated by using an imageacquisition condition (shooting condition information), information froman exposure state predicting unit 113 or an image recording medium 110,distance information acquired by a distance information acquiring unit105 a, or the like. The “image acquisition condition (shooting conditioninformation)” as used herein means a configuration of the image pickupunit 100, an exposure state determined by the aperture stop or the like,a focus position, a focal length of a zoom lens, or the like at the timewhen the analog signal is acquired. The image acquisition condition maybe acquired by a state detector 108 directly from a system controller111, and information on the image pickup unit 100 may be acquired from acontrol unit 107. In this embodiment, information on the configurationof the image pickup unit 100 is stored in the storage unit 109. Theexposure state predicting unit 113 predicts an exposure state observedin the shooting based on information from a photometry unit 112. Thedistance information acquiring unit 105 a acquires distance informationof an object space (an object) by using input parallax information(parallax information on the input image). The distance information isused to calculate a refocus control range.

The system controller 111 includes a display instructing unit 111 c. Thedisplay unit 106 switches “ON” and “OFF” of display and the displayimage based on a signal from the display instructing unit 111 c. Forinstance, when the image pickup apparatus 10 includes a release button,the display instructing unit 111 c displays a display signal of thedisplay image while the user pushes the release button down to a firstposition. In this situation, pushing the release button down to a secondposition deeper than the first position causes the image pickupapparatus to perform the shooting. This embodiment, however, is notlimited to this and the display instructing unit 111 c may bealternatively configured to send the signal by other means.

The system controller 111 further includes an in-focus object specifyingunit 111 b which specifies an object to be focused by the image pickupunit 100. A focus control unit 107 a provided in the control unit 107drives a focus mechanism of the image pickup unit 100 based on a signalfrom the in-focus object specifying unit 111 b to focus a specifiedobject. That is, the control unit 107 (the system controller 111) drivesthe imaging optical system 101 to perform focus control. When a shootinginstructing unit 111 d performs the shooting, the control unit 107adjusts an exposure of the image pickup unit 100 based on informationfrom the photometry unit 112. In this operation, the image acquired bythe image pickup element 103 is input to the image processing unit 105over a path similar to that described above, subjected to predeterminedprocessing, and then stored in the image recording medium 110 such as asemiconductor memory in a predetermined format. At the same time, theimage acquisition condition in the shooting acquired from the statedetector 108 is also recorded. The image recorded in the image recordingmedium 110 may be an image which has been subjected to thereconstruction processing.

In displaying the image stored in the image recording unit 110 on thedisplay unit 106, the image processing unit 105 performs processingbased on the image acquisition condition (shooting conditioninformation) used in the shooting. As a result, the image reconstructedwith desired settings (a resolution, a viewpoint, a focus position, adepth of field, etc.) is displayed on the display unit 106. Asynthesized image resolution specifying unit 111 a specifies theresolution of the reconstructed image. Moreover, for higher speeddisplay, the reconstructed image may be displayed on the display unit106 not via the image recording medium 110, with the desired settingsbeing stored in the storage unit 109 in advance. While there are otherelements of the image pickup apparatus 10 than those described above, adescription thereof will be omitted since such elements do not serve asmain elements in this embodiment. The series of controls described aboveis performed by the system controller 111.

Next, the configuration of the image pickup unit 100 in this embodimentwill be described. The image pickup unit 100 has an arrangementillustrated in FIG. 2. The lens array 102 is arranged at an image-sideconjugate plane of the imaging optical system 101 with respect to theobject plane 201. In addition, the lens array 102 is configured suchthat an exit pupil of the imaging optical system 101 and the imagepickup element 103 are in an approximate conjugate relationship. The“approximate conjugate relationship” includes not only a strictconjugate relationship, but also a relationship that would be evaluatedas a substantial conjugate relationship.

Rays from the object plane 201 are incident on different pixels of theimage pickup element 103 via the imaging optical system 101 and the lensarray 102 depending on their positions and angles on the object plane201. This leads to acquisition of parallax image data (light fielddata). In this process, the lens array 102 (a pupil dividing unit)serves to prevent the rays that have passed through different positionson the object plane 201 from being incident on the same pixel. That is,the lens array 102 causes the rays from the same position on the objectplane 201 to be incident on pixels of the image pickup element 103different from each other depending on pupil regions of the imagingoptical system 101 through which the rays pass. As a result, the imagepickup element 103 acquires an image in which pixel groups acquired byshooting the same region on the object plane 201 from a plurality ofviewpoints are arranged.

Subsequently, referring to FIG. 6, a section of the image pickup unit100 will be described. FIG. 6 is a sectional view of the image pickupunit 100 in this embodiment. The imaging optical system 101 of FIG. 6 isa single focus lens. A focus lens unit IF is moved on an optical axis OAto perform focusing. While the lens array 102 is constituted by a solidsingle lens in this embodiment, applicable lenses are not limited tothis and the lens array 102 may include a plurality of lenses. Inaddition, the lens array 102 may alternatively use a liquid lens, aliquid crystal lens, a diffraction optical element, or the like.Moreover, while, in this embodiment, a small lens constituting the lensarray 102 has both faces that are convex shapes, one of the faces may bea flat face or an aspherical face.

Subsequently, the refocus processing will be described. The refocusingis described in detail in Literature “Fourier Slice Photography”(written by Ren Ng, 2005 ACM Trans. Graph. 24, see pages 735-744.).Therefore, it will be described briefly in this embodiment. A basicprinciple of the refocusing is common among the configurationsillustrated in FIGS. 2 to 5. In this embodiment, the configurationillustrated in FIG. 2 will be described as an example.

Since the pupil of the imaging optical system 101 is divided into ninein two dimensions (three in one dimension) in FIG. 2, nine images havingdifferent viewpoint are acquired. An image corresponding to a certaindivided pupil is hereinafter referred to as a “single viewpoint image”.The nine single viewpoint images have a parallax to each other. For thisreason, a relative positional relationship among objects on the imagesvaries depending on an object distance. When single viewpoint images aresynthesized such that a certain object overlaps, objects located atdifferent object distances are synthesized, being displaced with eachother. This displacement makes the objects located at the differentobject distances blur. The blurring that occurs in this process isdetermined by a pupil corresponding to the single viewpoint images usedfor the synthesis, and synthesizing all the nine single viewpoint imagesenables reproducing the blurring of the image shot using the imagingoptical system 101. An object that is to overlap in such singleviewpoint image synthesis is chosen arbitrarily. This makes it possibleto reproduce the image shot by focusing on an arbitrary object by usingthe imaging optical system 101. This series of process is a principle ofpost-shooting focus control, namely, of the refocusing.

Subsequently, referring to FIG. 11, a method of generating the singleviewpoint image in this embodiment will be described. FIG. 11 is anexplanatory diagram of generation of the single viewpoint image in thisembodiment and illustrates a relationship between the lens array 102 andthe image pickup element 103 of FIG. 2. Each dashed-line circleindicates a region of a pixel on which a ray that has passed through asmall lens is incident. While FIG. 11 corresponds to a case where smalllenses are arrayed in a grid pattern, applicable arrays of the smalllenses are not limited to this. For instance, such an array mayalternatively be an array having a six-fold symmetry (a honeycombstructure) or an array in which each small lens is minutely displacedfrom a regular array. Each oblique-line portion of FIG. 11 indicates apixel on which rays that have passed through the same pupil region ofthe imaging optical system 101 are incident. Therefore, extracting sucha pixel indicated by its corresponding oblique-line portion enablesgenerating a single viewpoint image observed when an object space isseen from the bottom of the pupil of the imaging optical system 101.Similarly, extracting pixels whose relative positions to respectivedashed-line circles are the same enables generating other singleviewpoint images as well.

Subsequently, a refocus control range within which a focus position canbe changed will be described. Since the refocusing is performed byoverlapping the single viewpoint images, it is impossible to refocus ablurred object in each single viewpoint image. The reason for this isthat the blurred images remain blurred because a high frequencycomponent cannot be acquired by overlapping them with each other. Thatis, the refocus control range depends on the divided pupil of theimaging optical system 101. Since a depth of field of each singleviewpoint image becomes deeper as the pupil is divided into smallerportions, the refocus control range becomes broader accordingly.However, the depth of field in each single viewpoint image and therefocus control range do not necessarily match. This is because therefocus control range changes depending on a resolution ratio betweenthe single viewpoint image and a reconstructed image acquired byreconstructing the single viewpoint images. For example, when aresolution of the reconstructed image is lower than that of an image ateach viewpoint, a sampling pitch of a space component in thereconstructed image is larger than that of each single viewpoint image.Therefore, the reconstructed image has a deeper depth of field than thatof each single viewpoint image and the reconstructed image has a broaderrefocus control range accordingly. Conversely, when the resolution ofthe reconstructed image is larger than that of each single viewpointimage, the reconstructed image has a refocus control range narrower thanthe depth of field of each single viewpoint image accordingly. The abovedescription shows that it is necessary to take a reconstructioncondition for a single viewpoint image into consideration in order toacquire an accurate refocus control range of the reconstructed image.

Next, a method of calculating the refocus control range of thereconstructed image will be described in detail. First, a depth of focuscorresponding to a depth of field of the reconstructed image will beconsidered. In this embodiment, symbol E denotes a size of a permissiblecircle of confusion for the depth of focus and symbol Au denotes asampling pitch of an angle component of a ray. In this case, a refocuscoefficient α_(±) is represented by the following Expression (1).

$\begin{matrix}{\alpha_{\pm} = \frac{1}{1 \pm {{ɛ/\Delta}\; u}}} & (1)\end{matrix}$

Where a distance between an image-side principal plane of the imagingoptical system 101 and an image-side conjugate plane of the imagingoptical system 101 with respect to the object plane 201 is S₂, animage-side refocus range represented by Expression (1), which isrepresented by a product of the refocus coefficient α_(±) and thedistance S₂, falls within a range between α₊s₂ and α⁻s₂. Consequently, arange conjugate to the imaging optical system 101 corresponds to arefocus control range that is an object-side refocus range. Asillustrated in FIG. 12, a center position of the refocus range is afocus position of the imaging optical system 101. In this case, aposition of the lens array 102 is the center position of the refocusrange. A relation represented by Expression (1) holds in any of theconfigurations illustrated in FIGS. 2 to 5.

FIG. 12 is an explanatory diagram of the refocus control range in thisembodiment. An “image-side refocus control range d_(refocus)” means arange conjugate to the object-side refocus control range via the imagingoptical system 101. Symbol Δy denotes a sampling pitch of atwo-dimensional intensity distribution of light, which is equal to apitch Δ_(LA) of the lens array 102 in the configuration illustrated inFIG. 2. Since a pixel pitch A of the image pickup element 103 issufficiently small with respect to an exit pupil distance P, Expression(1) can be approximated as represented by the following Expression (2).α_(±) s ₂ =s ₂ ∓NFΔy=s ₂ ∓NFΔ _(LA) =s ₂ ∓NFε  (2)

In Expression (2), the “exit pupil distance P” of the imaging opticalsystem 101 means a distance between an exit pupil plane of the imagingoptical system 101 and the image-side conjugate plane of the imagingoptical system 101 with respect to the object plane 201. In addition,symbol N denotes a one-dimensional division number of the pupil of theimaging optical system 101 and symbol F denotes an F number of theimaging optical system 101.

The image-side refocus control range d_(refocus) is represented by thefollowing Expression (3) by using Expression (2).d _(refocus)=2NFε  (3)

The image-side refocus control range d_(refocus) can be converted intothe object-side refocus control range D_(refocus) by determining therange conjugate with respect to the imaging optical system 101. Theconjugate range can be acquired by applying a formula to determine adepth of field.

Of the object-side refocus control range D_(refocus), a range from theobject plane 201 toward the image pickup apparatus 10 is defined asD_(near) and a range in a direction away from the object plane 201 isdefined as D_(far). In addition, a distance between the object plane 201and an object-side principal plane of the imaging optical system 101 isdefined as S₁ and a focal length determined when the imaging opticalsystem 101 forms an image at a position where the object distance isinfinity is defined as f. Each of the distance S₁ and the focal length fhas a positive sign regardless of its direction. In this situation, thefollowing Expression (4) is satisfied.D _(refocus) =D _(far) +D _(near)  (4)

In Expression (4), the range D_(far) and the range D_(near) arerepresented by the following Expressions (5) and (6), respectively.

$\begin{matrix}{D_{far} = \frac{\left( {f - s_{1}} \right)^{2} \times N\; F\; ɛ}{f^{2} + {\left( {f - s_{1}} \right) \times N\; F\; ɛ}}} & (5) \\{D_{near} = \frac{\left( {f - s_{1}} \right)^{2} \times N\; F\; ɛ}{f^{2} - {\left( {f - s_{1}} \right) \times N\; F\; ɛ}}} & (6)\end{matrix}$

Next, in this embodiment, a method of placing a plurality of arbitrarypoints located in an object space within a refocus control range will bedescribed with reference to FIGS. 16 to 18. FIGS. 16 to 18 areflowcharts illustrating shooting processing in this embodiment. Theflowcharts of FIGS. 16 to 18 are different in terms of presence orabsence of arbitrary objects.

The flowchart of FIG. 16 is a flowchart illustrating an operation inwhich the arbitrary objects are present. This flowchart is, for example,as illustrated in FIGS. 19A and 19B, a flowchart illustrating anoperation in which a user chooses, at the time of shooting, at least oneof a plurality of objects, such as persons and objects, actually presentin the object space that the user desires to be included in the refocuscontrol range (or that the image pickup apparatus 10 automaticallychooses).

On the other hand, the flowcharts of FIGS. 17 and 18 are flowchartsillustrating an operation in which a specific object is absent. For easeof understanding, a description will be given using operation examples.First, an operation, as an example, used in the flowchart of FIG. 17will be described. As illustrated in FIGS. 20A and 20B, this operationis used when the user desires to set, as the refocus control range, arange from one meter in front of a focus position as a reference set bythe focus mechanism of the image pickup apparatus 10 (an actual value ofthe range is set arbitrarily) up to a position as far as possible. Forexample, this operation is used for changing a focus position if theuser shoots a player, as an object, who is moving at a high speed, suchas that observed in a game of sports, with other objects, such as aball, other players, and the like that are accidentally included in abackground of the object to be shot, being unintentionally focused. Inthis case, an actual object may be absent at a position located onemeter in front of the focus position since other objects than the actualobject to be shot are not necessarily included in the backgroundthereof. Conversely, a similar flowchart is used also when the userdesires to set, as the refocus control range, a range from a positionlocated several-meter far from the focus position (an actual value ofthe range is set arbitrarily) as a reference up to as near as possible.

Subsequently, the operation example used in the flowchart of FIG. 18will be described. As illustrated in FIGS. 21A and 21B, this operationis used when the user desires to set, as the refocus control range, arange from the focus position to a position as far as possible withoutincluding a range in front of the focus position. One application ofthis operation is a case where the user shoots a goal scene of atrack-and-field race (e.g., a 100-meter race) in front, with the firstplace runner being focused by the focus mechanism, and changes anoriginal focus position to another focus position to be focused on thesecond place runner or the third place runner after the shooting. Inthis case, it is not necessary to make the shot image refocusablebecause no object is present in front of the focus position originallyset (i.e., the first place runner). As described above, this operationcan be used when the user desires to effectively control the refocuscontrol range. Each step of the flowcharts of FIGS. 16 to 18 areperformed mainly by the image processing unit 105 based on a command (aninstruction) of the system controller 111.

First, referring to the flowchart of FIG. 16, a description will begiven of a case where the arbitrary objects are present in the shooting.At step S101, the image processing unit 105 acquires parallax images(information on the parallax images) acquired by the image pickup unit100. Subsequently, at step S102, the image processing unit 105 displaysa reconstructed image of the parallax images acquired at step S101 onthe display unit 106. This enables the user to check in real time theimage (a through-the-lens image) currently acquired by the image pickupelement 103. A focus position at this time point is, for example, afocus position focused by the focus mechanism, which is a reconstructedimage acquired without performing the refocus processing for theacquired parallax images. The image processing unit 105 mayalternatively generate a single viewpoint image whose viewpoint isclosest to a center of the pupil of the imaging optical system 101 anddisplay the single viewpoint image on the display unit 106. Simplerprocessing required for outputting a single viewpoint image than thatrequired for outputting a reconstructed image results in a less timelag, leading to a more speedy display. In addition, when an output is asingle viewpoint image, a viewpoint of the single viewpoint image may bean arbitrary position of the pupil of the imaging optical system 101.

Next, at step S103, a first focus position is specified based on theimage displayed at step S102. The first focus position may be specifiedby the user with, for example, a touch panel. Alternatively, the firstfocus position may be automatically specified utilizing a facerecognition technique or the like by the image pickup apparatus 10.Subsequently, at steps S104 and S105, the image processing unit 105 (thedistance information acquiring unit 105 a) acquires distance informationon a distance from the image pickup apparatus 10 to the first focusposition. In this process, first, at step S104, the image processingunit 105 prepares for acquiring the distance information at step S105.The “distance information” at step S105 means a distance from the imagepickup apparatus 10 to the first focus position.

A method of determining the distance from the image pickup apparatus 10to the first focus position depends on a method of focusing. Asexamples, cases of contrast AF and manual focusing (MF) will bedescribed. The contrast AF, which is referred to also asmountain-climbing AF, is a method of automatically driving the focusmechanism based on a contrast of the image acquired by the image pickupapparatus 10 to perform the focusing. On the other hand, the MF is amethod in which the user operates the focus mechanism to determine afocus position. In these methods of focusing, a movement distance of thefocus mechanism can be utilized for determining the distance from theimage pickup apparatus 10 to the first focus position. Depending onspecifications of the imaging optical system 101 in the image pickupunit 100, the movement distance of the focus mechanism corresponding toa distance from the image pickup apparatus 10 to an arbitrary focusposition is determined. The movement distance can be calculated eachtime because it is determined by a geometric optical calculation.Alternatively, the movement distance may be determined by previouslystoring a table of a relation between the movement distance of the focusmechanism and a distance from the image pickup apparatus 10 to thearbitrary focus position and referring to the table. In this case,operating the focus mechanism until it focuses on the first focusposition at step S104 enables the image processing unit 105 to acquirethe distance information on the distance from the image pickup apparatus10 to the first focus position at step S105.

When a phase difference AF is used as a method of focusing, step S104 isnot necessarily required. The “phase difference AF” means a method ofdetermining a distance by which the focus mechanism is moved by usingparallax images and does not need to actually move the focus mechanismin order to determine the distance. As long as the distance by which thefocus mechanism is moved is known, the distance from the image pickupapparatus 10 to an arbitrary focus position can be determined asdescribed above. In this case, step S104 may be omitted. As describedabove, while a process for acquiring the distance information isslightly different depending on the method of focusing, applicablemethods of acquiring the distance information are not limited to methodswhich use the focus mechanism. For instance, the distance informationmay be acquired by using a method such as DFD (Depth From Defocus),which is a conventional technique or the like, or a ranging unit thatutilizes an infrared ray or the like. In a case of any method offocusing is adopted at step S104, the focusing may be performed by usingthe focus mechanism such that the user can check the object located atthe first focus position.

Next, at step S106, the image processing unit 105 calculates a refocuscontrol range for the first focus position based on the distanceinformation acquired at step S105. As described above, the refocuscontrol range varies depending on a resolution ratio between each singleviewpoint image and its reconstructed image. For ease of description, acase where the resolution ratio is one will be considered. When theimage-side refocus control range falls within the range represented byExpression (2), this means that a refocusable region has been acquired.Therefore, a distance d_(r) between the image-side conjugate plane ofthe imaging optical system 101 with respect to the object plane 201 andthe image-side refocus control range only has to satisfy the followingExpression (7). Symbol d_(r) denotes a distance whose sign is positiveirrespective of a direction.d _(r) ≦NFε  (7)

It is found with reference to FIG. 27 that Expression (7) geometricallyrepresents NF=σ/Δ. FIG. 27 is an optical arrangement diagram of theimage pickup unit 100 illustrated in FIG. 2 and illustrates arelationship among parameters. In FIG. 27, symbol σ denotes a distancebetween the image-side principal plane of the lens array 102 and theimage pickup element 103. Each dashed line of FIG. 27 indicates a regionof the image pickup element 103 corresponding to one of the small lensesand each pixel indicated by an oblique line represents an insensitivezone on which no ray is incident. In this embodiment, since the lensarray 102 is configured so as not to have such an insensitive zone, therelation of Δ_(LA)=NΔ is satisfied. This embodiment, however, is notlimited to this and such an insensitive zone may be present. When thesize of the permissible circle of confusion, which defines a focallength, is characterized by the sampling pitch Δy=Δ_(LA) of the spacecomponent, Expression (7) is rewritten as the following Expression (8).

$\begin{matrix}{\frac{d_{r}}{{NF}\;\Delta_{LA}} = {\frac{d_{r}\Delta}{\Delta_{LA}\sigma} \leq \frac{ɛ}{\Delta_{LA}}}} & (8)\end{matrix}$

Next, a general case where a resolution ratio of the single viewpointimage and that of the reconstructed image is different will beconsidered. An angle of field of the reconstructed image and that of thesingle viewpoint image used for the reconstruction are equal to eachother. For this reason, when the resolution ratios are different fromeach other, values of their sampling pitches Δy are different from eachother. Generally, the smaller the value of sampling pitches Δy, thelarger the permissible circle of confusion is, and vice versa.Therefore, Expression (8) represents a ratio between the values of theirsampling pitches Δy of the single viewpoint image and the reconstructedimage and can be extended as represented by the following Expression(9).

$\begin{matrix}{\frac{d_{r}\Delta}{\;{\Delta_{LA}\sigma}} \leq {\frac{ɛ}{\Delta_{LA}}\sqrt{\frac{R_{mono}}{R_{synth}}}}} & (9)\end{matrix}$

In Expression (9), symbol R_(mono) denotes a resolution of the singleviewpoint image used for the synthesis and symbol R_(synth) denotes thatof the reconstructed image. A ratio of the sampling pitches Δy can beacquired by determining a square root of a ratio between the resolutionR_(mono) and the resolution R_(synth). It is found with reference toFIG. 27 that the resolution R_(mono) of the single viewpoint image isrepresented by the following Expression (10).

$\begin{matrix}{R_{mono} = {\left( \frac{\Delta}{\Delta_{LA}} \right)^{2}R_{total}}} & (10)\end{matrix}$

In Expression (10), symbol R_(total) denotes the number of effectivepixels of the image pickup element 103. Based on results of Expressions(9) and (10), the following Expression (11) that the image-side refocuscontrol range should satisfy is determined.

$\begin{matrix}{0.0 < {\frac{d_{r}}{\sigma}\sqrt{\frac{R_{synth}}{R_{total}}}} \leq 10.0} & (11)\end{matrix}$

Setting the image-side refocus control range as a range of Expression(11) enables acquiring a region refocusable after the shooting. A resultof Expression (11) cannot be a negative value in theory. In addition,when the result of Expression (11) is zero, this means that the focuscontrol cannot be performed. Therefore, it is impossible to exceed alower limit of Expression (11). An upper limit of the result ofExpression (11) represents a point spread of the reconstructed image atits focus position. The smaller the point spread, the sharper refocusingcan be performed. In a range beyond the upper limit of Expression (11),the point spread is wide making the image blurring even at its focusposition. That is, this means that the refocusing has failed.

Desirably, a sharper reconstructed image can be acquired by setting therefocus control range within a range of the following Expression (11a).

$\begin{matrix}{0.0 < {\frac{d_{r}}{\sigma}\sqrt{\frac{R_{synth}}{R_{total}}}} \leq 6.0} & \left( {11a} \right)\end{matrix}$

More desirably, a further sharper in-focus image can be acquired bysetting the refocus control range within a range of the followingExpression (11b).

$\begin{matrix}{0.0 < {\frac{d_{r}}{\sigma}\sqrt{\frac{R_{synth}}{R_{total}}}} \leq 3.0} & \left( {11b} \right)\end{matrix}$

Values of Expression (11) in this embodiment are as indicated inTable 1. In this embodiment, the number of effective pixels of the imagepickup element 103 is R_(total)=46.7×10⁶ (pix) and a distance betweenthe image-side principal plane of the lens array 102 and the imagepickup element 103 is c=0.0374 (mm). Symbol pix denotes a unitrepresenting the number of pixels. A pixel pitch of the image pickupelement 103 is Δ=0.0043 (mm) and that of the lens array 102 isΔ_(LA)=0.0129 (mm). A focal length, an F number, and the number ofone-dimensional pupil divisions of the imaging optical system 101 aref=14.0 (mm), F=2.9, and N=3, respectively. The resolution R_(synth) ofthe reconstructed image may be selected by the synthesized imageresolution specifying unit 111 a from the following three resolutions:8.0×10⁶ pix, 5.2×10⁶ pix, and 2.0×10⁶ pix. A value of the distance d_(r)for each resolution is as indicated in Table 1. Since a resolution for asingle viewpoint image is 5.2×10⁶ pix, it is required to increase aresolution by means of, for example, super-resolution from subpixelshift in order to generate a reconstructed image having a resolution of8.0×10⁶ pix. The resolution R_(synth) of the reconstructed image may beother value than the above values and is not limited to the threevariations. However, in such a case, the distance d_(r) is determined soas to satisfy Expression (7).

The method of calculating a refocus control range described above,however, requires a large amount of processing when calculating therefocus control range on an as needed basis in the shooting. In order toprevent this, the storage unit 109 may be alternatively configured topreviously store a table of a refocus control range for each imageacquisition condition and to read corresponding data instead ofcalculating the refocus control range.

As another method of acquiring a refocus control range, a method can beemployed which actually generates a refocused reconstructed image andevaluates a contrast and the like of an object present at a focusposition. This method, however, requires generating the reconstructedimage while changing the focus position and determining, on an as neededbasis, whether or not the refocusing can be performed. This makes theentire processing time-consuming. Moreover, when an object is notpresent at a refocused focus position, an accurate focus control rangecannot be acquired because it is impossible to evaluate the contrast orthe like of the object. Thus, it is desirable to use the above-mentionedmethod to acquire the refocus control range.

The description of the flowchart of FIG. 16 will now be continued. Atstep S107, a second focus position is specified. Similarly to the firstfocus position, the second focus position can be specified by the userwith the touch panel or the like in the shooting. Alternatively, thesecond focus position may be automatically specified by using the facerecognition technology or the like by the image pickup apparatus 10.Since subsequent steps S108 and S109 are similar to steps S104 and 105,respectively, except that the second focus position is applicable, adetailed description thereof will be omitted.

Next, at step S110, the image processing unit 105 determines, by usingthe following Expression (12), whether or not the second focus positionis included in the refocus control range of the first focus positionfirstly specified.s _(1st) _(_) _(obj) −D _(1st) _(_) _(near) ≦D _(2nd) _(_) _(obj) ≦s_(1st) _(_) _(obj) +D _(1st) _(_) _(far)  (12)

In Expression (12), symbol s_(1st) _(_) _(obj) denotes a distancebetween the first focus position and the object-side principal plane ofthe imaging optical system 101, and symbols D_(1st) _(_) _(near) andD_(1st) _(_) _(far) denote an object-side refocus control range locatedin front of and behind the first focus position, respectively. When thesecond focus position is included in the refocus control range of thefirst focus position at step S110 (if the determination made at stepS110 is “OK”), the image processing unit 105 waits until the shooting isperformed or specifies a larger number of focus positions. When theimage processing unit 105 specifies the larger number of focus positions(n-th focus position), it is enough to return to step S107 to repeatoperations of step S107 to S111. In this embodiment, for ease ofunderstanding, a description will be given of a case where two focuspositions, which are the first and second focus positions, arespecified.

If the determination made at step S110 is “NG”, the control unit 107drives the focus mechanism to adjust the focus position and then changesthe refocus control range in step S200. The image processing unit 105determines a third focus position such that both of the first and secondfocus positions are included in the refocus control range. A method ofdetermining the third focus position will be described later.

After determining the third focus position, the image processing unit105 waits for a next instruction at step S111. Upon receipt of ashooting instruction at step S112, the control unit 107 drives the focusmechanism to the third focus position and then changes a current focusposition to the third focus position in step S113. At step S114, thesystem controller 111 performs the shooting. If no determination as “NG”is made at step S110, the first and third focus positions are equal toeach other. In this situation, the focus mechanism moves the focusposition to the first focus position (keeps the first focus positionwhen the focus mechanism has already moved to the focus position at stepS104) and then performs the shooting. The above process is a series ofoperations in the case where the arbitrary objects are present at thetime of the shooting.

Next, a description will be given of a method of determining the thirdfocus position such that both of the first and second focus positionsare included in the refocus control range. In this embodiment, threespecific methods will be described. A first method is a method whichdetermines an optimum focus position based on a table of relationshipbetween the object distance and the refocus control range. A secondmethod is a method which gradually moves the focus mechanism to searchfor the optimum focus position. A third method is a method whichsearches for the optimum focus position based on a contrast of a singleviewpoint image. These methods correspond to step S200 of FIG. 16 andare illustrated in flowcharts of FIGS. 23 to 25.

The first method is a method which derives the third focus positionbased on the table of the relationship between the object distance andthe refocus control range previously stored. FIG. 22A illustrates anexample of the table in this embodiment. FIG. 23 is a flowchartillustrating the first method. At steps S105 and S109, the distancesfrom the image pickup apparatus 10 to the first focus position and thesecond focus position (distance information) are acquired. Based on thedistance information, the image processing unit 105 calculates arelative distance between the first focus position and the second focusposition (relative distance information) at step S211. Based on arequired refocus control range determined by this calculation, the imageprocessing unit 105 subsequently refers to the table illustrated in FIG.22A at step S212 and calculates the third focus position at step S213.Since only discrete data can be indicated on the table, interpolationprocessing may alternatively be performed to calculate a solution whenthe table does not indicate the solution. When a solution is not presentin which both of the first and second focus positions are included inthe refocus control range, the flow moves to step S300.

The image processing unit 105 acquires the relative distance informationbetween the first focus position and the second focus position in thismanner. Then, the image processing unit 105 calculates the third focusposition by using the table which indicates a relationship between theobject distance and the refocus control range. It is preferable that theimage pickup apparatus 10 further includes a storage unit (e.g., thestorage unit 109) storing the table which indicates the relationshipbetween the object distance and the refocus control range.

The second method is a method which finds out a solution in a searchmanner by gradually moving the focus mechanism. FIG. 24 is a flowchartillustrating the second method. First, a description will be givenassuming that, for ease of understanding, the first focus position isgiven at step S221 as an initial value of the third focus position to becalculated. This embodiment, however, is not limited to this and anotherposition such as the second focus position may be used as the initialvalue.

Subsequently, at step S222, the image processing unit 105 calculates acurrent refocus control range. After that, at step S223, the imageprocessing unit 105 determines whether or not the first and second focuspositions are included in the refocus control range by using Expression(12). If the determination is “OK” (the first and second focus positionsare included in the refocus control range), the flow returns to stepS111 of FIG. 16. On the other hand, if this determination is “NG” (thefirst and second focus positions are out of the refocus control range),the control unit 107 drives the focus mechanism by a certain amount atstep S224. When the initial value is the first focus position, it isenough to drive the focus mechanism in a direction in which an in-focusstate is acquired toward the second focus position. When the initialvalue is neither the first focus position nor the second focus position,it is enough to drive the focus mechanism by a certain amount such thatthe in-focus state is acquired toward farther one of the first andsecond focus positions. The control unit 107 may gradually drive thefocus mechanism in a certain direction and calculate a refocus controlrange at every movement. Alternatively, the control unit 107 may roughlydrive the focus mechanism at the beginning and then reverse a movementdirection of the focus position to change a movement amount of the focusposition when the second focus position is included in the refocuscontrol range, but the first focus position is out of the refocuscontrol range. If the image processing unit 105 fails to acquire asolution by moving the focus mechanism by whatever distance, the flowtransfers to step S300.

As described above, the image processing unit 105 calculates the refocuscontrol range (for a plurality of focus positions different from eachother) while changing a focus position. When the first and second focuspositions are included in the refocus control range at a specific focusposition, the image processing unit 105 sets the specific focus positionas the third focus position. Preferably, the system controller 111 (thecontrol unit 107) determines a movement direction and a movement amountof the focus position based on the refocus control range calculatedwhile changing the focus position.

The third method is a method which makes a determination based on acontrast value of a single viewpoint image. As described above, therefocus control range depends on the depth of field of a singleviewpoint image (strictly speaking, however, a resolution ratio betweenthe single viewpoint image and the reconstructed image needs to beconsidered). For this reason, when comparing the contrasts of theobjects at positions corresponding to the first and second focusposition in a single viewpoint image and determining that the contrastvalues are a certain threshold level or higher, the image processingunit 105 determines that the first and second focus positions areincluded in the refocus control range. This method thus requires as aprecondition that there are objects at both of the first and secondfocus positions. The contrast value can be calculated using thefollowing Expression (13) where, for example, a maximum brightness valuein an arbitrary region (e.g., 10×10 pixels) is defined as L_(max) and aminimum brightness value in the arbitrary region is defined as L_(min).m=(L _(max) −L _(min))/(L _(max) +L _(min))  (13)

Each of the brightness values L_(max) and L_(min) in Expression (13) maybe either single color of R, G or B, or a white brightness valuecombined by a certain weighing. When a value of m determined byExpression (13) is not smaller than an arbitrary threshold, the imageprocessing unit 105 determines that the objects are included in therefocus control range. The threshold itself is, however, differentdepending on specifications of the image pickup apparatus 10 and thelike (however, since the resolution ratio between the single viewpointimage and the reconstructed image needs to be considered in fact, it ispreferable to set the threshold according the resolution ratio).

FIG. 25 is a flowchart illustrating the third method. Since, in thethird method, steps other than those at which a contrast value iscalculated and compared (steps S231 and 234) are similar to those of thesecond method described above, a description thereof will be omitted. Atstep S232, the image processing unit 105 calculates contrast values ofobjects located at the first and second focus positions. Then, at stepS233, when the contrast values of the first focus position and thesecond focus position are not larger than the threshold (“NG” at stepS233), the control unit 107 drives the focus mechanism by a certainamount at step S234.

As described above, the image processing unit 105 calculates thecontrast values at the first focus position and the second focusposition with respect to the single viewpoint image. When the contrastvalues of the first focus position and the second focus position are notsmaller than a predetermined threshold, the image processing unit 105sets a specific focus position as the third focus position.

While the three methods of calculating the third focus position havebeen described above, other step such as a check step may be providedbetween each step. Moreover, part or all of the three methods may becombined.

Next, step S300 in a case where a solution for the third focus positionis not present will be described. In this case, the display instructingunit 111 c gives the display unit 106, such as a finder and a backmonitor of the image pickup apparatus 10, a warning to notify the userthat a chosen object is out of the refocus control range.

FIGS. 26A and 26B are diagrams illustrating an example of a shootingscene in this embodiment. FIG. 26A illustrates an example of providing awarning on a screen with letters and FIG. 26B illustrates an example ofblinking the chosen object to notify the user that the chosen object isout of the refocus control range. It is preferable to provide a step fornotifying the user that a solution is not present. Instead of providingsuch a message on the display image, other notification method such asvoice may be used.

Next, the flowchart of FIG. 17 will be described. Since steps S401 toS403 of FIG. 17 are similar to steps S101 to S103 of the flowchart ofFIG. 16, respectively, a description thereof will be omitted. At stepS404, the control unit 107 drives the focus mechanism to move a currentfocus position to the focus position specified at step S403.Subsequently, at step S405, the distance information acquiring unit 105a acquires distance information between the image pickup apparatus 10and the first focus position. Since a method of acquiring the distanceinformation is similar to that at step S105 described above, adescription thereof will be omitted. Subsequently, at step S406, theimage processing unit 105 acquires a refocus control range at the firstfocus position. Since a method of acquiring and calculating the refocuscontrol range is similar to that at step S106, a description thereofwill be omitted.

Next, at step S407, the image processing unit 105 specifies the secondfocus position. This example includes the case where one meter before(or far from) the first focus position is set as the second focusposition as described in the case of the shooting of the scene of thesport game. The second focus position may be specified at the time ofthe shooting or may be preset in the storage unit 109 in the imagepickup apparatus 10 and read at this step.

Subsequently, at step S408, the image processing unit 105 determineswhether or not the second focus position is included in the refocuscontrol range acquired at step S406. A method of determining this issimilar to that at step S110. When the second focus position is includedin the refocus control range, the image processing unit 105 displays animage currently acquired by the image pickup element 103 (thethrough-the-lens image) (similar to step S402) and waits for a shootinginstruction. Upon receipt of the shooting instruction such as that givenby the user by pushing of the release button, the control unit 107drives the focus mechanism to adjust the focus position at step S411.After that, the system controller 111 performs the shooting at step S412to finish this flow. Whether to focus either of the first focus positionor the second focus position at step S411 depends on the determinationmade at step S408. When the flow proceeds to step S409 withoutexperiencing any “NG”, it is no problem with the first focus position.Therefore, the focus position observed at step S411 is equal to thefirst focus position. When the flow proceeds to step S200 after thedetermination made at step S408 is “NG”, in step S411, the control unit107 changes the focus position by the focus mechanism to the third focusposition calculated at step S200. Steps S200 and S300 are as describedabove.

Next, the flowchart of FIG. 18 will be described. However, since stepsS501 to S505 are similar to steps S401 to S405 of the flowchart of FIG.17, respectively, a description thereof will be omitted. At step S506,the control unit 107 changes the focus position to the third focusposition by driving the focus mechanism so as to move the focus positionby a certain amount. This example is a case where, as described with theshooting of the goal scene, the first focus position is located near aboarder of the front-side refocus control range and the refocus controlrange is set as far from the front-side refocus control range aspossible, i.e. is set to the back side (the front side and the back sidemay be opposite to each other). Therefore, at this step, as illustratedin FIG. 28, a finally determined focus position of the imaging opticalsystem. 101 becomes the third focus position (D_(3rd)). FIG. 28 is anexplanatory diagram of the refocus control range in this embodiment.

Similarly, a boundary of the front-side refocus control range (the sideof the image pickup apparatus 10) with respect to the third focusposition is the first focus position D_(1st), and a distance between thethird focus position D_(3rd) and the first focus position D_(1st) isdenoted as D_(3rd) _(_) _(near). Conversely, a boundary of the back-siderefocus control range (in a direction away from the image pickupapparatus 10) with respect to the third focus position D_(3rd) is thesecond focus position D_(2nd) and a distance between the third focusposition D_(3rd) and the second focus position D_(2nd) is denoted asD_(3rd) _(_) _(far). The first focus position D_(1st) and the secondfocus position D_(2nd) are required to satisfy relations represented bythe following Expressions (14) and (15) for ensuring this situationwhere a distance between the third focus position D_(3rd) and theobject-side principal plane of the imaging optical system 101 is definedas s₁ _(_) _(3rd).D _(1st) =s ₁ _(_) _(3rd) −D _(3rd) _(_) _(near)  (14)D _(2nd) =s ₁ _(_) _(3rd) +D _(3rd) _(_) _(far)  (15)

In Expression (14), however, since the first focus position D_(1st) islocated at or near an edge of the refocus control range, it may be outof the refocus control range depending on an accuracy of the focusmechanism. For this reason, it is preferable that Expression (14) has acertain margin α (constant) as in the following Expression (14a).D _(1st) =s ₁ _(_) _(3rd) −D _(3rd) _(_) _(near)+α  (14a)

In Expression (14a), the constant α may be arbitrarily determined by theuser so as to suit to a shooting scene or may be written in the storageunit 109 or the like from the beginning. At step S506, the focusposition is changed to the third focus position D_(3rd) that satisfiesthe relations described above. There are several possible methods ofchanging the focus position. When a method of referring to the table ofthe relationship between the focus positions and the refocus controlranges, such as the first method, is used, it is enough to determine thethird focus position D_(3rd) based on data read from the table or dataacquired by performing the interpolation processing for reference data.Even when the table is not available, it is possible to calculate thethird focus position D_(3rd) which satisfies Expression (14) or (14a) bycalculating the refocus control range based on the above-mentionedparameters. It is preferable to calculate the third focus positionD_(3rd) by the above method because the focus position can be changedquickly. In this case, at step S507, the image processing unit 105determines whether or not the calculated third focus position satisfiesExpression (14) or (14a).

When there is no problem at this step, the image processing unit 105causes the through-the-lens image to be displayed and waits for theshooting instruction at step S508. Upon receipt of the shootinginstruction, the control unit 107 proceeds from step S509 to step S510and changes the focus position to the determined third focus position.The system controller 111 then performs the shooting at step S111. Thisflow ends upon completion of the shooting. On the other hand, if thedetermination is “NG” at step S507, an error such as a failure tocalculate the third focus position D_(3rd) in the calculation processmay have occurred. In this case, it is enough to change the focusposition by switching the method to a search approach described later(step S600) or to directly proceed to step S300 and give the user awarning.

Next, a description will be given of a method of determining the thirdfocus position D_(3rd) in a search manner without using the table of therelationship between the focus positions and the refocus control ranges(step S600). This method sets an appropriate initial value as a focusposition change amount and then searches for the third focus positionD_(3rd) therefrom. While this method resembles the second method and thethird method described above, its flowchart is slightly different.

FIGS. 29 and 30 are flowcharts of the shooting processing in thisembodiment and correspond to step S600 of FIG. 18. FIG. 29 correspondsalso to the above-mentioned second method and FIG. 30 illustrates theflowchart corresponding to the third method.

First, at step S506 of FIG. 18, the control unit 107 changes the focusposition by using the appropriate initial value. A direction in whichthe focus position is changed is a direction toward the far side (backside) of the first focus position (a direction away from the imagepickup apparatus 10). After that, the image processing unit 105 makes adetermination at step S507. If this determination is “OK”, the flowproceeds to step S508 as described above and subsequent processing whichis the same as that described above follows. On the other hand, if thedetermination is “NG”, the flow proceeds to step S600.

First, referring to FIG. 29, step S600 at which the second method isused will be described in detail. The flow proceeds to step S611 afterthe determination is “NG” at step S507. The control unit 107 againchanges the focus position by a certain amount at this step. The amountof change in the focus position may be the same amount as that describedabove or may be changed. Subsequently, at step S612, the imageprocessing unit 105 calculates the refocus control range at the changedfocus position. A method of calculating the refocus control range issimilar to the above-mentioned method. After that, at step S613, theimage processing unit 105 makes a determination similar to that at stepS507. If the determination is “OK”, the flow proceeds to step S508. Onthe other hand, if the determination is “NG”, the flow returns to stepS611 and the control unit 107 repeats to change the focus position. Amethod of moving the focus position at step S611 may gradually move thefocus position in a certain direction and calculate the refocus controlrange in each movement. Alternatively, the method may roughly move thefocus position at the beginning and search for the first focus positionby reversing a movement direction of the focus position so as to changea movement amount of the first focus position if the first focusposition becomes out of the refocus control range. If an error occurs atstep S611 such as a deviation from a range (a mechanical range) withinwhich the focus mechanism can move the focus position, the flow proceedsto step S300.

Next, referring to FIG. 30, step S600 using the third method will bedescribed in detail. The flow proceeds to step S621 after thedetermination is “NG” at step S507. In this step, the control unit 107again moves the focus position by a certain amount. The amount of changein the focus position may be the same amount as that described above ormay be changed. Subsequently, at step S622, the image processing unit105 calculates a contrast value of each object at the changed focusposition. Since a method of calculating the contrast value is similar tothe third method, a description thereof will be omitted. After that, atstep S623, the image processing unit 105 makes a determination similarto that at step S507. If the determination is “OK”, the flow proceeds tostep S508. On the other hand, if the determination is “NG”, the flowreturns to step S621 and the control unit 107 repeats to change thefocus position. A method of moving the focus position at step S621 maygradually move the focus position as described in step S611 or may movethe focus position in a search manner. If an error occurs at step S621such as a deviation from a range (a mechanical range) within which thefocus mechanism can move the focus position, the flow proceeds to stepS300.

In this embodiment, the image processing unit 105 acquires informationon the refocus control range. Then, the system controller 111 (thecontroller 107) performs the focus control based on the information onthe refocus control range. Preferably, the image processing unit 105determines whether or not the second focus position is included in therefocus control range for the first focus position. The systemcontroller 111 (the control unit 107) performs the focus control whenthe second focus position is not included in the refocus control range.The system controller 111 then changes the first focus position to thethird focus position such that both of the first and second focuspositions are included in the refocus control range.

Preferably, the image processing unit 105 acquires the third focusposition such that the first and second focus positions are included inthe refocus control range based on the information on the refocuscontrol range. More preferably, the system controller 111 (the controlunit 107) performs the focus control so as to move the focus position tothe third focus position. More preferably, the system controller 111outputs a warning to the user when it determines that the third focusposition cannot be set.

More preferably, the image processing unit 105 changes the refocuscontrol range according to at least one of shooting conditioninformation, an angle-of-field region of an input image, and aresolution of a reconstructed output image.

According to this embodiment, it is possible to provide an image pickupapparatus capable of, after a shooting, acquiring a refocus image with afocus position intended by a user.

Second Embodiment

Next, an image pickup apparatus in the second embodiment of the presentinvention will be described. The image pickup apparatus of thisembodiment has a basic configuration illustrated in FIG. 1 similarly tothe image pickup apparatus of the first embodiment. In addition, animage pickup unit 100 of this embodiment is arranged as illustrated inFIG. 3. FIG. 7 is a sectional view of the image pickup unit 100 of thisembodiment. An imaging optical system 101 is a zoom lens. The imagingoptical system 101 is constituted by, in order from an object side, afirst lens unit L1 having a positive refractive power, a second lensunit L2 having a positive refractive power, a third lens unit L3 havinga negative refractive power, a fourth lens unit L4 having a positiverefractive power, and a fifth lens unit L5 having a positive refractivepower. In varying the magnification, the first lens unit L1 and thefifth lens unit L5 are fixed, and the second lens unit L2, the thirdlens unit L3, and the fourth lens unit L4 are moved on an optical axis.In the focusing, the second lens unit L2 is driven.

As illustrated in FIG. 3, a lens array 102 is arranged closer to theobject than to an image-side conjugate plane 202 of the imaging opticalsystem 101 with respect to an object plane 201. The image-side conjugateplane 202 and an image pickup element 103 are arranged to have aconjugate relationship via the lens array 102. Rays from the objectplane 201 pass through the imaging optical system 101 and the lens array102, and then are incident on different pixels of the image pickupelement 103 depending on their positions and angles on the object plane201, which leads to acquisition of a light field. In configurationsillustrated in FIGS. 3 and 4, the image pickup element 103 acquires animage in which a plurality of small images with different shootingviewpoints and different shooting ranges from each other are included.The configuration illustrated in FIG. 4 is similar to that illustratedin FIG. 3 except that the lens array 102 is arranged closer to the imageside than to the image-side conjugate plane 202. What is different fromthe configuration illustrated in FIG. 3 is that the lens array 102 viewsthe image formed by the imaging optical system 101 as a real object andcauses the image to be re-formed on the image pickup element 103.However, the configurations illustrated in FIGS. 3 and 4 aresubstantially the same because the lens array 102 views the image formedby the imaging optical system 101 as the real object and causes theimage to be formed on the image pickup element 103 in both of theconfigurations. Therefore, the following description holds also for theconfiguration illustrated in FIG. 4.

Next, refocus processing in this embodiment will be described. Therefocus processing of this embodiment is qualitatively similar to thatof the first embodiment, and thus it is enough to overlap images fromdivided pupils of the imaging optical system 101 with a displacementcorresponding to a distance to an object to be focused.

FIGS. 10A and 10B are explanatory diagrams of generation of the refocusimage and illustrate detailed partial views of the lens array 102 andthe image pickup element 103 in the configuration illustrated in FIG. 3.In this embodiment, the lens array 102 is constituted by small lenseseach having a flat object-side face and a convex image-side face.Applicable shapes of the small lenses are, however, not limited to thissimilarly to the first embodiment. Each dashed-dotted line in FIGS. 10Aand 10B indicates an angle of field of each small lens. Projecting andsynthesizing each pixel value acquired by the image pickup element 103on a virtual imaging plane 203 via the small lens corresponding to thepixel enables generating a refocus image focused on the virtual imagingplane 203.

The “virtual imaging plane 203” as used herein means a plane conjugate,via the imaging optical system 101, to a plane at the side of an objectto be focused by the refocusing. For instance, in order to generate animage focused on the object plane 201 of FIG. 3, it is enough to set thevirtual imaging plane 203 to the image-side conjugate plane 202.

In FIGS. 10A and 10B, the pixels projected in the generation of therefocus image are indicated by a dashed-line and, for ease ofunderstanding, are depicted to be shifted from each other instead ofoverlapping them. The refocus image may be generated by a method ofmoving each pixel in parallel to synthesize the pixels when the pixelsoverlap similarly to the above-mentioned method of projecting eachpixel. In this generation, when a region of the lens array 102 throughwhich a light beam incident on pixels has passed is the same, parallelmovement amounts of the pixels are the same. That is, an operation ofthe pixels in the generation of the refocus image in FIGS. 3 and 4 isdetermined depending on the region of the lens array 102 through whichthe light beam has passed.

Next, the refocus control range will be described. The refocus controlrange in this embodiment is represented by Expression (1) similarly tothe first embodiment. A relationship thereof is as illustrated in FIG.13. FIG. 13 is an explanatory diagram of the refocus control range inthis embodiment. Symbol Δy in FIG. 13 denotes a sampling pitch of atwo-dimensional intensity distribution of light in FIGS. 3 and 4, andthe relation of Δy=Δσ₁/σ₂ is satisfied. This is because the lens array102 causes the image formed by the imaging optical system 101 to beimaged on the image pickup element 103 in a reduced magnification ofσ₂/σ₁ by viewing the image as an imaginary object.

In this situation, symbol σ₁ denotes a distance between the image-sideconjugate plane 202 and the object-side principal plane of the lensarray 102 and symbol σ₂ denotes a distance between the image-sideprincipal plane of the lens array 102 and the image pickup element 103.Since the relation of Δ<<P is satisfied also in this embodiment,Expression (1) can be approximated to Expression (2). In thisembodiment, a process to include a plurality of arbitrary points in anobject space in the refocus control range is indicated by the flowchartsof FIGS. 16 to 18. Therefore, a description of parts similar to those ofthe first embodiment will be omitted.

At steps S106 (FIG. 16) and S406 (FIG. 17) or at step S507 (FIG. 18),the refocus control range of a reconstructed image is calculated andacquired. This calculation method is similar to that of the firstembodiment and calculates the image-side refocus control range. FIG. 22Billustrates an example of a table of this embodiment in a case where thetable is used by the first method described above. As can be seen fromFIG. 13, the geometrical relation of NF=σ₁/Δ_(LA) is satisfied. Inaddition, since the relation of Δy=Δσ₁/σ₂ is satisfied as describedabove, the following Expression (16) is satisfied.

$\begin{matrix}{R_{mono} = {\left( \frac{\sigma_{2}}{\sigma_{1}} \right)^{2}R_{total}}} & (16)\end{matrix}$

The use of these leads to determination of the following Expression (17)that the distance d_(r) should satisfy.

$\begin{matrix}{0.0 < {\frac{\Delta_{LA}d_{r}}{{\Delta\sigma}_{1}}\sqrt{\frac{R_{synth}}{R_{total}}}} \leq 10.0} & (17)\end{matrix}$

Mathematical meanings of upper and lower limits in Expression (17) aresimilar to those in Expression (11).

Desirably, a sharpness of the reconstructed image increases by includingthe limits within a range of the following Expression (17a).

$\begin{matrix}{0.0 < {\frac{\Delta_{LA}d_{r}}{{\Delta\sigma}_{1}}\sqrt{\frac{R_{synth}}{R_{total}}}} \leq 6.0} & \left( {17a} \right)\end{matrix}$

More desirably, a sharper reconstructed image can be acquired byincluding the limits within a range of the following Expression (17b).

$\begin{matrix}{0.0 < {\frac{\Delta_{LA}d_{r}}{{\Delta\sigma}_{1}}\sqrt{\frac{R_{synth}}{R_{total}}}} \leq 3.0} & \left( {17b} \right)\end{matrix}$

A value of Expression (17) in this embodiment is as indicated inTable 1. In this embodiment, the number of effective pixels of the imagepickup element 103 is R_(total)=150.0×10⁶ (pix). In addition, therelations of σ₁=0.3712 (mm) and σ₂=0.0740 (mm) are satisfied. A pixelpitch of the image pickup element 103 is Δ=0.0024 (mm) and a pitch ofthe lens array 102 is Δ_(LA)=0.0256 (mm). A focal length at a wide-angleend of the imaging optical system 101, a focal length at a telephoto endof the imaging optical system 101, an F number from the wide-angle endto the telephoto end, and the number of one-dimensional pupil divisionsare f_(W)=72.2 (mm), f_(T)=194.0 (mm), F=2.9, and N=5, respectively. Thesynthesized image resolution specifying unit 111 a can choose aresolution R_(synth) of the reconstructed image from three resolutionsof 10.0×10⁶ pix, 6.0×10⁶ pix, and 3.0×10⁶ pix. The distance d_(r) foreach resolution is as indicated in Table 1. Since a resolution persingle viewpoint image is 6.0×10⁶ pix, a resolution enhancement by asuper-resolution from subpixel shift or the like is required in order togenerate a reconstructed image having a resolution of 10.0×10⁶ pix.

According to this embodiment, it is possible to provide an image pickupapparatus capable of, after a shooting, acquiring a refocus image with afocus position intended by a user.

Third Embodiment

Next, an image pickup apparatus in the third embodiment of the presentinvention will be described. The image pickup apparatus of thisembodiment has a basic configuration illustrated in FIG. 1 similarly tothe image pickup apparatus of the first embodiment. An image pickup unit100 of this embodiment is arranged as illustrated in FIG. 5 and isarranged as illustrated in FIG. 15 when seen from an object side. In thethird embodiment, the image pickup unit 100 includes a plurality ofoptical systems 101 a to 101 g (imaging optical systems) each having apositive refractive power and is configured to have a six-fold symmetrywith an optical axis of the optical system 101 b being a rotationalaxis. That is, assuming that a pupil of the imaging optical systems iscreated by synthesizing (combining) pupils of the plurality of opticalsystems 101 a to 101 g (pupil dividing units), the plurality of opticalsystems 101 a to 101 g are arranged such that rays from the sameposition on an object plane are incident on pixels of image pickupelements different from each other, depending on a pupil region of theimaging optical system through which the rays pass.

This embodiment is, however, not limited to this and the number and thearray of the imaging optical systems can be changed. At the image sideof the plurality of optical systems 101 a to 101 g, image pickupelements 103 a to 103 g are arranged. However, the number of the imagepickup elements is not required to be plural and may be singular as longas images formed by the optical systems 101 a to 101 g can be acquired.

The rays refracted by the plurality of optical systems 101 a to 101 gare received by the corresponding image pickup elements 103 a to 103 g.Images acquired by the image pickup elements 103 a to 103 g are parallaximages acquired by observing an object space from different viewpoints.Synthesizing the plurality of images enables acquiring a light field ofthe object space.

FIG. 8 is a sectional view of the imaging optical system (optical system101 a) and the image pickup element 103 a in this embodiment. The otherimaging optical systems (optical systems 101 b to 101 g) and imagepickup elements 103 b to 103 g are similar to this. Configurations ofthe imaging optical systems 101 a to 101 g may, however, be differentfrom each other. The imaging optical system (optical system 101 a) ofFIG. 8 is a single focus lens. Focus control can be performed bychanging a distance between the optical system 101 a and the imagepickup element 103 a.

Refocus processing in this embodiment can be performed by overlappingimages at viewpoints with a displacement corresponding to a distance toan object to be focused, similarly to that in the first embodiment. Afocus control range within which the refocusing can be performed is alsorepresented by Expression (1). A relationship thereof is as illustratedin FIG. 14. FIG. 14 is an explanatory diagram of the refocus controlrange in this embodiment. In this embodiment, the relations of Δy=Δ andΔu=P_(mono)/F_(mono) are satisfied. In the latter relation, symbolF_(mono) denotes an F number of one of the imaging optical systems 101 ato 101 g and symbol P_(mono) denotes an exit pupil distance of one ofthe imaging optical systems 101 a to 101 g. Since the relation ofΔ<<P_(mono) is satisfied, Expression (1) can be approximated to thefollowing Expression (18).α_(±) s ₂ =s ₂ ∓F _(mono)Δ_(y) =s ₂ ∓F _(mono)Δ  (18)

A process to include a plurality of arbitrary points in the object spacewithin the refocus control range is illustrated by the flowcharts ofFIGS. 16 to 18 and a description for parts similar to those of the firstembodiment will be omitted. At step S106 (FIG. 16), S406 (FIG. 17), orS507 (FIG. 18), the refocus control range of the reconstructed image iscalculated and acquired. An image-side refocus control range iscalculated based on a theory similar to that of the first embodiment.

The refocus control range is acquired by substituting the F number ofone of the imaging optical systems 101 a to 101 g acquired by aprediction of an exposure state to the F number F_(mono). Where aresolution of an image formed by any of the imaging optical systems 101a to 101 g that has the F number F_(mono) is R_(mono), the followingExpression (19) which the distance d_(r) should satisfy is acquired.

$\begin{matrix}{0.0 < {\frac{d_{r}}{F_{mono}\Delta}\sqrt{\frac{R_{synth}}{R_{mono}}}} \leq 10.0} & (19)\end{matrix}$

Mathematical meanings of upper and lower limits in Expression (19) aresimilar to those in Expression (11).

Desirably, a sharpness of the reconstructed image increases by includingthe limits within a range of the following Expression (19a).

$\begin{matrix}{0.0 < {\frac{d_{r}}{F_{mono}\Delta}\sqrt{\frac{R_{synth}}{R_{mono}}}} \leq 6.0} & \left( {19a} \right)\end{matrix}$

More desirably, a further sharper reconstructed image can be acquired byincluding the limits within a range of the following Expression (19b).

$\begin{matrix}{0.0 < {\frac{d_{r}}{F_{mono}\Delta}\sqrt{\frac{R_{synth}}{R_{mono}}}} \leq 3.0} & \left( {19b} \right)\end{matrix}$

A value of Expression (19) in this embodiment is as indicated inTable 1. In this embodiment, the number of effective pixels and pixelpitch of each of the image pickup elements 103 a to 103 g areR_(mono)=19.3×10⁶ (pix) and Δ=0.0012 (mm), respectively. A focal lengthand a full open F number of each of the imaging optical systems 101 a to101 g are f=50.0 (mm) and F=1.8, respectively. In Table 1, an F numberin the shooting is F_(mono)=1.8. In the case of a different F number,the distance d_(r) is determined so as to satisfy Expression (19). Inaddition, in a high angle-of-field region, the refocus control range ischanged depending on a vignetting of the light beam. For instance, whena focal length in an angle-of-field region in an image is as twice aslong with respect to a region on an axis, a refocus control range in theangle-of-field region is widen twice as wide as that on the axis. Theresolution R_(synth) of the reconstructed image may be selected by thesynthesized image resolution specifying unit 111 a from the followingthree resolutions of 19.3×10⁶ pix, 10.0×10⁶ pix, and 5.0×10⁶ pix. Adistance d_(r) for each resolution is as indicated in Table 1.

According to this embodiment, it is possible to provide an image pickupapparatus capable of, after a shooting, acquiring a refocus image with afocus position intended by a user.

Fourth Embodiment

Next, referring to FIGS. 31 to 33, a basic configuration of an imageprocessing system in the fourth embodiment of the present invention willbe described. FIG. 31 is a block diagram of an image processing system30 in this embodiment. FIG. 32 is a schematic configuration diagram ofthe image processing system 30. FIG. 33 is a block diagram of a singleviewpoint image acquiring unit 400 in this embodiment.

As illustrated in FIG. 32, an image pickup unit 300 includes a pluralityof single viewpoint image acquiring units 400 a to 400 d. A singleviewpoint image acquiring unit 400 illustrated in FIG. 33 is one of thesingle viewpoint image acquiring units 400 a to 400 d. The singleviewpoint image acquiring units 400 a to 400 d each has qualitativelythe same configuration as that illustrated in FIG. 5.

An image processing unit 301 illustrated in FIG. 31 is a computer devicewhich performs processing illustrated in FIGS. 16 to 18 and includes adistance information acquiring unit 301 a similarly to the firstembodiment. An image processed by the image processing unit 301 isoutput to at least one of a display unit 302, a recording medium 303,and an output unit 304. The display unit 302 is, for example, a liquidcrystal display or a projector. The recording medium 303 is, forexample, a semiconductor memory, a hard disk, or a server on a network.The output unit 304 is a printer or the like. The user can works whilechecking the image via the display unit 302 in the shooting or editing.The image processing unit 301 has a function to perform developmentprocessing and other image processing as needed in addition to theprocessing illustrated in FIGS. 16 to 18 and the reconstructionprocessing. A system controller 305 such as a personal computer (a PC)controls each member. A storage medium such as a CD-ROM storing aprogram which causes the computer to perform this control mayalternatively be used. The system controller 305 includes, similarly tothe first embodiment, a synthesized image resolution specifying unit 305a, an in-focus object specifying unit 305 b, a display instructing unit305 c, a shooting instructing unit 305 d, and a focus control unit 305e.

In FIG. 33, an image formed on an image pickup element 403 via animaging optical system 401 is converted into an electrical signal (ananalog signal). The analog signal is converted by an A/D converter 404into a digital signal. The digital signal is subjected to predeterminedprocessing by an image processing unit 405 and then output to eachmember in the single viewpoint image acquiring unit 400 and to the imageprocessing unit 301. Upon receipt of a signal from the system controller305, a system controller 411 controls each member of the singleviewpoint image acquiring unit 400. An exposure state predicting unit413 is a member which predicts an exposure state in the shooting basedon information from a photometry unit 412. A display unit 406 switches“ON” and “OFF” of display and a display image via the image processingunit 405, based on a signal from the system controller 411. When theshooting is performed according to a command (an instruction) from thesystem controller 411, a control unit 407 adjusts an exposure of theimaging optical system 401 based on the information from the photometryunit 412. At this time, the image acquired by the image pickup element403 is input via a path similar to that described above to the imageprocessing unit 405, subjected to predetermined processing, and thenstored in an image recording medium 410 such as a semiconductor memoryin a predetermined format. At the same time, an image acquisitioncondition in the shooting acquired from a state detector 408 is alsorecorded. Furthermore, the image recorded in the image recording medium410 may be an image which has been subjected to the reconstructionprocessing. Moreover, for higher speed display, the reconstructed imageas the image recorded in the image recording medium 410 may be displayedon the display unit 406 not via the image recording medium 410, withdesired settings being stored in a storage unit 409 in advance.

FIG. 9 is a sectional view of an imaging optical system 401 a and animage pickup element 403 a of the single viewpoint image acquiring unit400 a. The imaging optical system 401 a illustrated in FIG. 9 is asingle focus lens and drives a focus lens unit IF to perform thefocusing. The other single viewpoint image acquiring units 400 b to 400d are similarly configured. However, the single viewpoint imageacquiring units 400 a to 400 d may have configurations different fromeach other and the number and array thereof are not limited.

Refocus processing of this embodiment is similar to that of the thirdembodiment, and generation of a display image in the shooting andediting is also similar to that of the third embodiment. In addition, avalue of Expression (19) is as indicated in Table 1. In this embodiment,the number of effective pixels and a pixel pitch of each of the imagepickup elements 403 a to 403 d are R_(mono)=32.0×10⁶ (pix) and Δ=0.0052(mm), respectively. A focal length and a full open F number of each ofthe imaging optical systems 401 a to 401 d are f=200.0 (mm) and F=2.0,respectively. The values in Table 1 are values calculated with apredicted F number in the shooting being set to F_(mono)=2.0. Theresolution R_(synth) of the reconstructed image may be selected by thesynthesized image resolution specifying unit 305 a from the followingthree resolutions of 64.0×10⁶ pix, 32.0×10⁶ pix, and 8.0×10⁶ pix. Thedistance d_(r) for each resolution is as indicated in FIG. 1. Aresolution enhancement by the super-resolution from subpixel shift orthe like is required in order to generate a reconstructed image having aresolution of 64.0×10⁶ pix.

According to this embodiment, it is possible to provide an imageprocessing system capable of, after a shooting, acquiring a refocusimage with a focus position intended by a user.

The present invention can be implemented also by performing thefollowing processing. That is, the processing is that software (aprogram) implementing functions of the embodiments described above isprovided via a network or various storage media to a system or anapparatus and a computer (or a CPU or an MPU) thereof reads out andexecutes the program.

According to the embodiments, it is possible to provide an image pickupapparatus capable of, after a shooting, acquiring a refocus image with afocus position intended by a user, a method of controlling the imagepickup apparatus, and a non-transitory computer-readable storage medium.

TABLE 1 FIRST R_(total) (pix) σ (mm) EMBODIMENT 46.7 × l0⁶  0.0374R_(synth) (pix) d_(r) (mm) Expression (11)  8.0 × 10⁶ 0.2260 2.5  5.2 ×10⁶ 0.6166 5.5  2.0 × 10⁶ 1.7174 9.5 SECOND R_(total) (pix) Δ (mm)Δ_(LA) (mm) σ₁ (mm) EMBODIMENT 150.0 × 10⁶  0.0024 0.0256 0.3712R_(synth) (pix) d_(r) (mm) Expression (17) 10.0 × 10⁶ 1.3208 9.8  6.0 ×10⁶ 0.9918 5.7  3.0 × 10⁶ 0.6398 2.6 THIRD R_(mono) (pix) Δ (mm)F_(mono) EMBODIMENT 19.3 × 10⁶ 0.0012 1.8 R_(synth) (pix) d_(r) (mm)Expression (19) 19.3 × 10⁶ 0.0060 2.8 10.0 × 10⁶ 0.0171 5.7  5.0 × 10⁶0.0407 9.6 FOURTH R_(mono) (pix) Δ (mm) F_(mono) EMBODIMENT 32.0 × 10⁶0.0052 2.0 R_(synth) (pix) d_(r) (mm) Expression (19) 64.0 × 10⁶ 0.01622.2 32.0 × 10⁶ 0.0187 1.8  8.0 × 10⁶ 0.0249 1.2

OTHER EMBODIMENTS

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions recorded on a storage medium (e.g., non-transitorycomputer-readable storage medium) to perform the functions of one ormore of the above-described embodiment(s) of the present invention, andby a method performed by the computer of the system or apparatus by, forexample, reading out and executing the computer executable instructionsfrom the storage medium to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or more ofa central processing unit (CPU), micro processing unit (MPU), or othercircuitry, and may include a network of separate computers or separatecomputer processors. The computer executable instructions may beprovided to the computer, for example, from a network or the storagemedium. The storage medium may include, for example, one or more of ahard disk, a random-access memory (RAM), a read only memory (ROM), astorage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2013-171816, filed on Aug. 22, 2013, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image pickup apparatus capable of processingan input image to generate a plurality of output images with focuspositions different from each other, the image pickup apparatuscomprising: an imaging optical system; an image pickup element includinga plurality of pixels; a lens array configured such that rays from thesame position on an object plane are incident on pixels of the imagepickup element different from each other depending on a pupil region ofthe imaging optical system through which the ray passes; a control unitconfigured to drive the imaging optical system to perform focus control;and an image processing unit configured to acquire a first focusposition and a second focus position, determine a third focus positionbased on the acquired first focus position and the acquired second focusposition such that a refocus control range associated with the thirdfocus position includes the first focus position and the second focusposition, and perform image processing for the input image acquired bythe image pickup element to generate the output image, wherein the imageprocessing unit is configured to determine the third focus position by:acquiring relative distance information between the first focus positionand the second focus position, and determining the third focus positionby using a table that indicates a relationship between an objectdistance and the refocus control range, and wherein the control unit isconfigured to perform the focus control based on the determined thirdfocus position.
 2. The image pickup apparatus according to claim 1,wherein the image processing unit is configured to determine whether therefocus control range with respect to the first focus position includesthe second focus position, and wherein, in a case where the second focusposition is not included in the refocus control range, the control unitis configured to drive the imaging optical system such that both of thefirst focus position and the second focus position are included in therefocus control range, to perform the focus control.
 3. The image pickupapparatus according to claim 1, wherein the control unit is configuredto drive the imaging optical system so as to move the focus position tothe third focus position to perform the focus control.
 4. The imagepickup apparatus according to claim 1, further comprising a storage unitstoring the table that indicates the relationship between the objectdistance and the refocus control range.
 5. The image pickup apparatusaccording to claim 1, wherein the image processing unit is configuredto: calculate the refocus control range while changing the focusposition, and set a specific focus position as the third focus positionwhen the first focus position and the second focus position are includedin the refocus control range with respect to the specific focusposition.
 6. The image pickup apparatus according to claim 5, whereinthe control unit is configured to determine a movement direction and amovement amount of the focus position based on the refocus control rangecalculated by the image processing unit while changing the focusposition.
 7. The image pickup apparatus according to claim 1, whereinthe image processing unit is configured to: calculate contrast values atthe first focus position and the second focus position with respect to asingle viewpoint image, and set a specific focus position as the thirdfocus position when each of the contrast values at the first focusposition and the second focus position is not less than a predeterminedthreshold.
 8. The image pickup apparatus according to claim 1, whereinthe image processing unit includes a distance information acquiring unitconfigured to acquire distance information of an object, and the imageprocessing unit is configured to calculate the refocus control range byusing the distance information.
 9. The image pickup apparatus accordingto claim 8, wherein the distance information acquiring unit isconfigured to acquire the distance information by using parallaxinformation of the input image.
 10. The image pickup apparatus accordingto claim 1, wherein the image processing unit is configured to changethe refocus control range according to shooting condition information.11. The image pickup apparatus according to claim 1, wherein the imageprocessing unit is configured to change the refocus control rangedepending on an angle-of-field region of the input image.
 12. The imagepickup apparatus according to claim 1, wherein the image processing unitis configured to change the refocus control range depending on aresolution of the processed output image.
 13. An image pickup apparatuscapable of processing an input image to generate a plurality of outputimages with focus positions different from each other, the image pickupapparatus comprising: an imaging optical system including a plurality ofoptical systems; at least one image pickup element including a pluralityof pixels; a control unit configured to drive the imaging optical systemto perform focus control; and an image processing unit configured toacquire a first focus position and a second focus position, determine athird focus position based on the acquired first focus position and theacquired second focus position such that a refocus control rangeassociated with the third focus position includes the first focusposition and the second focus position, and generate the output imagefrom the input image acquired by the image pickup element, wherein whena pupil of the imaging optical system is a pupil formed by combiningpupils of the plurality of optical systems, the plurality of opticalsystems are arranged such that rays from the same position on an objectplane are incident on pixels of the image pickup element different fromeach other depending on a pupil region of the imaging optical systemthrough which the ray passes, wherein the image processing unit isconfigured to determine the third focus position by: acquiring relativedistance information between the first focus position and the secondfocus position, and determining the third focus position by using atable that indicates a relationship between an object distance and therefocus control range, and wherein the control unit is configured toperform the focus control based on the determined third focus position.14. A method of controlling an image pickup apparatus capable ofprocessing an input image to generate a plurality of output images withfocus positions different from each other, the method comprising:acquiring the input image, the input image including information for alight field, taken by an image pickup apparatus which includes animaging optical system and an image pickup element including a pluralityof pixels; acquiring a first focus position and a second focus position;determining a third focus position based on the acquired first focusposition and the acquired second focus position such that a refocuscontrol range associated with the third focus position includes thefirst focus position and the second focus position; and driving theimaging optical system based on the determined third focus position toperform focus control, wherein determining the third focus positionincludes acquiring relative distance information between the first focusposition and the second focus position, and determining the third focusposition by using a table that indicates a relationship between anobject distance and the refocus control range.
 15. A non-transitorycomputer-readable storage medium storing a program configured to cause acomputer to execute a method of controlling an image pickup apparatuscapable of processing an input image to generate a plurality of outputimages with focus positions different from each other, the methodcomprising: acquiring the input image, the input image includinginformation for a light field, taken by an image pickup apparatus whichincludes an imaging optical system and an image pickup element includinga plurality of pixels; acquiring a first focus position and a secondfocus position; determining a third focus position based on the acquiredfirst focus position and the acquired second focus position such that arefocus control range associated with the third focus position includesthe first focus position and the second focus position; and driving theimaging optical system based on the determined third focus position toperform focus control, wherein determining the third focus positionincludes acquiring relative distance information between the first focusposition and the second focus position, and determining the third focusposition by using a table that indicates a relationship between anobject distance and the refocus control range.